NAVAL  POSTGRADUATE  SCHOOL 
Monterey,  California 


THESIS 


WORLD  WIDE  WEB  ADMINISTRATION  AND 
DISTRIBUTION  OF  THE  MAINTENANCE  CLIMATE 
ASSESSMENT  SURVEY  (MCAS) 

by 

Jeff  W.  Fenton 
September  1999 

Thesis  Advisor:  John  K.  Schmidt 

Co- Advisor:  Douglas  E.  Brinkley 

Second  Reader:  Robert  R.  Read 


Approved  for  public  release;  distribution  is  unlimited. 


I>TIC  QUALITY  INSPECTED  4  19991126  103 


REPORT  DOCUMENTATION  PAGE 

Form  Approved.  \ 
OMBNo.  0704-0188 

1  Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instruction, 
searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send 
comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burden,  to 
Washington  headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA 
22202-4302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188)  Washington  DC  20503. 

1.  AGENCY  USE  ONLY  (Leave  blank)  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES  COVERED 

September  1999  Master’s  Thesis 

4.  TITLE  AND  SUBTITLE  : 

World  Wide  Web  Administration  and  Distribution  of  the  Maintenance  Climate 

Assessment  Survey  (MCAS) 

5.  FUNDING  NUMBERS 

6.  AUTHOR® 

Fenton,  Jeff  W. 

7.  PERFORMING  ORGANIZATION  NAME®  AND  ADDRESSEES) 

Naval  Postgraduate  School 

Monterey,  CA  93943-5000 

8.  PERFORMING 
ORGANIZATION  REPORT 
NUMBER 

9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

N/A 

10.  SPONSORING/ 
MONITORING 

AGENCY  REPORT 

NUMBER 

11.  SUPPLEMENTARY  NOTES 

The  views  expressed  in  this  thesis  are  those  of  the  author  and  do  not  reflect  the  official  policy  or  position  of  the  Department  of  Defense  or  the  U.S. 
Government. 

12a.  DISTRIBUTION /AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 

12b.  DISTRIBUTION  CODE 

13.  ABSTRACT  (maximum  200  words) 

Naval  Aviation  has  been  tasked  to  reduce  it’s  1996  human  factors  related  Class  A  flight  mishap  rate  in  half 
by  the  year  2000.  The  Human  Factors  Quality  Management  Board  (HFQMB)  was  established  in  1996  to  help  Naval 
Aviation  achieve  its  mishap  reduction  goals.  Previous  HFQMB  mishap  reduction  initiatives  focused  on  flight  mishaps 
and  aircrew  related  issues,  however  a  recent  shift  to  broaden  the  scope  now  puts  attention  on  aircraft  maintenance  and 
consequently  maintainers.  Baker  (1998)  developed  from  a  Maintainer’s  perspective  a  Climate  Assessment  Survey 
(MCAS).  The  purpose  of  the  MCAS  was  to  provide  a  diagnostic  tool  to  identify  potential  intervention  areas  from  the 
perspective  of  maintenance-related  mishaps.  This  thesis  seeks  to  provide  a  vehicle  fostering  the  proliferation  of  the 
MCAS  throughout  the  fleet  by  adapting  it  for  the  World  Wide  Web  (WWW),  and  developing  a  prototype  Web  site.  A 
usability  analysis  was  conducted  to  form  a  basis  for  future  modification  to  the  online  version  of  the  MCAS,  with  the 
ultimate  goal  being  an  effective  tool  to  promote  the  reduction  of  human  factors  related  mishaps  in  Naval  Aviation 
maintenance. 

14.  SUBJECT  TERMS 

Aviation  Safety,  Aviation  Maintenance  Safety,  Maintenance  Command  Assessment  Survey. 

15.  NUMBER 

OF  PAGES 

100 

16.  PRICE 

CODE 

17.  SECURITY 

CLASSIFICATION  OF  REPORT 
Unclassified 

18.  SECURITY  CLASSIFICATION 
OF  THIS  PAGE 

Unclassified 

19.  SECURITY 

CLASSIFICATION  OF 

ABSTRACT 

Unclassified 

20. 

LIMITATION 

OF  ABSTRACT 

UL 

NSN  7540-01-280-5500 


Standard  Form  298  (Rev.  2-89) 


Approved  for  public  release;  distribution  is  unlimited 


WORLD  WIDE  WEB  ADMINISTRATION  AND 
DISTRIBUTION  OF  THE  MAINTENANCE  CLIMATE 
ASSESSMENT  SURVEY  (MCAS) 

Jeff  W.  Fenton 

Lieutenant  Commander,  United  States  Navy 
B.S.,  Indiana  University,  1983 

Submitted  in  partial  fulfillment  of  the 
requirements  for  the  degree  of 

MASTER  OF  SCIENCE  IN  INFORMATION  TECHNOLOGY  MANAGEMENT 

from  the 

NAVAL  POSTGRADUATE  SCHOOL 
September  1999 


Author: 


(\l 

Robert  R.  Read,  Second  Reader 


iii 


IV 


ABSTRACT 


Naval  Aviation  has  been  tasked  to  reduce  its  1996  human  factors  related  Class  A 
flight  mishap  rate  in  half  by  the  year  2000.  The  Human  Factors  Quality  Management 
Board  (HFQMB)  was  established  in  1996  to  help  Naval  Aviation  achieve  its  mishap 
reduction  goals.  Previous  HFQMB  mishap  reduction  initiatives  focused  on  flight 
mishaps  and  aircrew  related  issues,  however  a  recent  shift  to  broaden  the  scope  now  puts 
attention  on  aircraft  maintenance  and  consequently  maintainers.  Baker  (1998)  developed, 
from  a  Maintainer’s  perspective  a  Climate  Assessment  Survey  (MCAS).  The  purpose  of 
the  MCAS  was  to  provide  a  diagnostic  tool  to  identify  potential  intervention  areas  from 
the  perspective  of  maintenance-related  mishaps.  This  thesis  seeks  to  provide  a  vehicle 
fostering  the  proliferation  of  the  MCAS  throughout  the  fleet  by  adapting  it  for  the  World 
Wide  Web  (WWW),  and  developing  a  prototype  Web  site.  A  usability  analysis  was 
conducted  to  form  a  basis  for  future  modification  to  the  online  version  of  the  MCAS, 
with  the  ultimate  goal  being  an  effective  tool  to  promote  the  reduction  of  human  factors 
related  mishaps  in  Naval  Aviation  maintenance. 
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I.  INTRODUCTION 


A.  HUMAN  FACTORS  QUALITY  MANAGEMENT  BOARD 

1.  Overview 

Following  the  tragic  January  1996  Nashville,  Tennessee  Navy  F-14  mishap  that 
killed  the  aircrew  and  several  local  residents.  Vice  Admiral  Bennitt,  the  then  Commander 
of  the  Naval  Air  Force,  U.  S.  Pacific  Fleet,  created  a  Human  Factors  Quality  Management 
Board  (HFQMB)  to  improve  aviation  safety  through  identifying  strategies  to  prevent 
human  error.  The  purpose  of  the  HFQMB  simply  stated  in  its  charter  (1996)  is: 

To  significantly  reduce  the  rate  of  occurrence  of  naval  aviation  mishaps 
caused  by  aircrew/operator  error  by  identifying  systemic  improvements  in 
the  processes  and  systems  that  affect  human  performance  and  guard 
against  human  error.  Further,  to  institutionalize  continual  improvement  in 
these  areas. 

Senior  Naval  Aviation  leaders  recognized  a  need  to  form  a  coalition  between  them 
as  policy  makers  and  individuals  who  work  directly  in  the  Naval  Aviation  community. 
Consequently,  the  HFQMB  is  composed  of  a  Flag  Officer  and  senior  operational  Officers 
from  both  the  Navy  and  Marine  Corps  including  a  cross  section  of  all  communities  and 
ranks  from  Squadron  Commanding  Officers  to  Fleet  Lieutenants.  The  implementation  of 
the  HFQMB  findings  was  intended  to  foster  a  reduction  in  the  1996  Naval  Aviation  class 
“A”  flight  mishap  rate  by  50  percent  within  three  years  and  by  75  percent  within  10  years 
(HFQMB  Charter,  1996). 
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2.  Human  Factors  Quality  Management  Board  Efforts 

The  HFQMB’s  strategy  of  taking  an  across  the  board  approach  to  improve  its 

processes  and  systems  encompassed  a  three-prong  approach,  including:  (1)  Mishap  Data 
analysis,  (2)  Benchmarking  Best  Procedures,  (3)  Climate  Safety  Assessment  (See  Figure 
1).  Mishap  Data  Analysis  included  the  practice  of  observing  patterns  that  may  have 
formed  in  past  mishaps,  and  trends  that  may  been  have  established  in  relation  to  human 
factors.  Bench  Marking  involved  observing  the  practices  used  in  other  organizations  to 
combat  human  factor  related  error.  The  Climate  Safety  Assessment  was  designed  to 
qualify  issues  that  might  impact  these  human  factor  errors. 


Mishap  Data 
Analysis 


Benchmarking 


Climate  Safety 
Assessment 


Brainstorming 


Recommendations 
for  Intervention 


Figure  1.  HFQMB  Methodology 
From  “AIRPAC  Brief,”  by  CDR  J.  Schmidt,  March  1998. 


The  first  area  of  interest  to  the  HFQMB,  analysis  of  mishap  data,  represents 
research  conducted  without  the  need  for  external  data  collection.  The  Naval  Safety 
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Center  (NSC)  maintains  a  database  of  mishaps  dating  back  20  years;  this  historical  data  is 
categorized  by  1050  characteristics  (NSC,  1999).  Taking  the  NSC  data  and  classifying 
the  cause  and  type  of  mishaps  enabled  aggregate  analysis  by  the  HFQMB.  The  NSC’s 
mishap  database  also  allowed  the  HFQMB  to  analyze  the  mishaps  with  enough  specificity 
that  when  coupled  with  the  appropriate  modeling  techniques,  enabled  meaningful  action 
development. 

Benchmarking  processes/practices  was  the  second  area  of  endeavor  by  the 
HFQMB  for  mishap  reduction.  Benchmarking  is  the  process  through  which  internal  of 
external  organizations'  performance  levels  are  compared,  and  its  processes/practices  are 
observed.  The  organizational  benchmarks  can  be  utilized  to  establish  guidelines  and 
ultimately  lead  to  "Best  Practice"  decisions  (Chapman,  1997).  In  this  instance,  the 
HFQMB  observed  processes  from  organizations  that  seek  the  same  safety  goals  as  Naval 
-  Aviation  such  as,  the  U.S.  Air  Force,  and  the  commercial  airlines,  etc.  (HFQMB,  1996). 

The  third  area  of  focus  by  the  HFQMB  is  addressed  by  the  Command  Safety 
Assessment  (CSA),  a  tool  designed  by  Ciavarelli  and  Figlock  (1997)  at  the  Naval 
Postgraduate  School  to  measure  safety  climate  in  a  squadron  from  the  aircrew 
perspective.  The  CSA  (Discussed  in  subsequent  chapters)  enabled  the  HFQMB  to 
measure  the  climate  and  process  effectiveness  within  an  organizations’  flight  operations. 
Recommendations  were  deemed  a  critical  part  of  the  overall  success  of  the  HFQMB ’s 
actions.  Using  the  information  acquired  through  the  means  of  observation  and  collection, 
the  HFQMB  prioritized  the  recommendations  in  terms  of  their  potential  to  achieve  the 
mishap  reduction  goals  and  forwarded  them  to  the  appropriate  authority  for  coordination 
and  implementation  at  the  organizational  level  of  Naval  Aviation  (HFQMB,  1996). 
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3.  Human  Factors  Quality  Management  Board  Results 

Historically,  the  mishap  rates  of  the  last  two  decades  have  been  low  by  past 
standards  (See  Figure  2).  However,  despite  a  low  mishap  rate,  it  is  important  to  realize 
that  Naval  Aviation  strikes  several  aircraft  per  year.  An  example  of  the  effect  lost  aircraft 
have  on  the  fleet  is  illustrated  by  the  data  taken  from  fiscal  year  1997  through  1998.  In 
FY  ‘97,  a  total  of  65  aircraft  were  accepted  into  the  Naval  Aviation  inventory  (Naval 
Aviation  News,  1998).  During  FY  ‘98,  36  aircraft  were  removed  from  the  inventory  as  a 
result  of  mishaps,  (NSC,  1999).  The  aircraft  lost  in  1998  represent  55  percent  of  those 
purchased  in  the  previous  year.  The  class  “A”  mishap  rate  has  been  essentially  on  a 
plateau  since  1988.  Without  a  change  in  leadership  action  or  the  conduct  of  Naval 
Aviation,  the  mishap  rate  would  not  be  expected  to  deviate  from  its  current  level. 

Contributing  to  mishap  data  analysis,  Shappell  and  Weigmann  (1995)  adapted 
Reason’s  (1990)  Human  Error  Model  to  re-classify  the  divisions  of  Naval  Aircrew  Flight 
mishaps.  This  classification  model  took  the  larger  category  of  human  error  and  expanded 
it  to  three  sub-areas  of  responsibility  (Nutwell  &  Sherman,  1997).  The  new  divisions  of 
human  error  consisted  of:  unsafe  aircrew  conditions,  unsafe  supervisory  conditions,  and 
unsafe  aircrew  acts.  The  redesigned  model  gave  a  better  understanding  of  the  human 
cause  factors  than  previous  classifications  in  the  traditional  model;  more  accurate 
reporting  resulted  in  improved  understanding,  and  clearly  indicated  the  appropriate 
intervention  areas  to  achieve  the  goals  sought  by  the  HFQMB.  Notably,  in  a  report  issued 
in  1 997  by  the  Department  of  the  Navy,  40  percent  of  the  procedure  violations  committed 
by  aircrew  were  conscious  breeches,  and  50  percent  of  the  supervisory  failures  were 
labeled  unsafe  supervisory  conditions.  Both  aircrew  and  supervisory  conditions  are  an 
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indication  of  climate  within  the  organization.  Relatively  little  work  has  been  completed 
on  organizational  factors  when  considering  the  effectiveness  of  a  squadrons  performance 
and  safety  (Ciavarelli  &  Figlock,  1997).  Program  and  policy  of  the  squadron  are  a  part 
the  safety  climate  and  culture  maintained  within  the  organization. 

Figure  2.  FY  50-99  Flight  Mishap  Rates. 

NAVAL  AVIATION  MISHAP  RATE 


From:  Naval  Safety  Center  1999.  (Through  30  JUN  99) 


The  efforts  of  the  HFQMB  were  expected  to_bring  about  fundamental  changes  in 
the  way  business  was  conducted  in  Naval  Aviation,  with  these  changes,  the  established 
mindset  regarding  attainable  mishap  reduction  would  have  to  change.  (HFQMB  Charter, 
1996).  The  recommendations  of  the  HFQMB  resulted  in  a  record  mishap  rate  for  the 
U.S.  Navy  in  1997,  with  a  mishap  rate  of  1 .29  %,  the  lowest  level  of  class  A  mishaps  ever 
(NSC,  1999).  The  U.  S.  Marine  Corps  reached  a  comparable  milestone  in  1998  with  the 
organizations  lowest  mishap  rate  of  2.52  %  (NSC,  1999).  The  past  actions  taken  by  the 
HFQMB  have  had  the  desired  effect  of  reducing  the  human  factor  related  mishap  rate.  To 
continue  to  reduce  the  human  factors  related  mishaps,  refinements  in  the  process 
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evaluation  would  need  to  take  place.  Nutwell  and  Sherman  (1997)  state  human  error 
represents  a  large  portion  of  the  causal  factors  that  contribute  to  Class  “A”  mishaps  and 
propose  additional  reductions  in  the  mishap  rate  could  be  attained  with  refinements  in  this 
area.  Traditional  approaches  targeting  aircrew  human  factors  should  be  expanded  to 
include  the  maintainer  with  potential  gains  in  mishap  reduction.  If  further  improvements 
in  mishap  reduction  were  to  occur  continued  exploration  of  all  facets  of  prevention 
needed  to  be  studied  and  understood.  Efforts  in  mishap  reduction  were  concentrated  in 
the  aircrew  side  of  human  factors  caused  mishaps,  expanding  the  focus  to  include  the 
maintenance  caused  human  factors  mishaps  could  enhance  the  overall  reduction  of  the 
mishap  rate.  The  desire  to  measure  the  climate  of  safety  from  a  maintainer’ s  viewpoint 
was  a  topic  that  necessitated  the  development  of  a  tool  for  this  purpose. 


B.  AVIATION  SAFETY  CLIMATE  ASSESSMENT 

1.  Command  Safety  Assessment 

The  Command  Safety  Assessment  (CSA)  was  developed  by  Ciavarelli  and 
Figlock  (1997)  of  the  Naval  Postgraduate  School;  its  goal  was  to  estimate  the 
effectiveness  of  organizational  safety  programs.  To  assess  the  effectiveness  of  safety 
programs  and  processes,  it  is  necessary  to  understand  the  climate  that  is  maintained 
within  the  organization.  Organizational  culture  is  defined  by  Merritt  and  Helmreich 
(1996)  in  the  following  manner. 
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Culture  can  be  defined  as  the  values,  beliefs,  rituals,  symbols  and 
behaviors  that  we  share  with  others  that  help  define  us  as  a  group, 
especially  in  relation  to  other  groups.  Culture  gives  us  cues  and  clues  on 
how  to  behave  in  normal  and  novel  situations,  thereby  making  the  world 
less  uncertain  and  more  predictable  for  us.  There  are  two  important  and 
distinct  components  of  culture.  The  surface  structure,  or  outer  layer  of 
culture  consists  of  observable  behaviors  and  recognizable  physical 
manifestations  such  as  member’s  uniforms,  signs  and  logos,  and 
documents.  The  deep  structure,  or  inner  layer  of  culture,  consists  of  the 
values,  beliefs  and  assumptions  which  underlie  the  surface  structure  and 
provide  the  logic  which  guides  the  members’  behaviors. 


Measuring  culture  and  looking  at  the  safety  climate  an  organization  fosters  is  part 
of  understanding  the  organization  and  the  people  that  make  up  the  group.  Each  person 
brings  their  history  and  multi-cultural  memberships  with  them  to  an  organization  (Merritt 
&  Helmriech,  1996).  If  organizational  culture  has  an  impact  on  the  safety  of  the  group,  it 
would  be  desirable  for  the  group  to  have  a  unified,  integrated  organizational  culture, 
rather  than  one  made  up  of  diverse  values.  Much  attention  has  been  focused  on  human 
error  in  complex  systems,  but  relatively  little  has  been  accomplished  on  the  potential 
contribution  that  organizational  factors  have  on  crew  performance  and  safety  (Ciavarelli 
&  Figlock,  1997). 

Work  in  the  area  of  climate  measurement  within  Naval  Aviation  squadrons  is 
built  on  research  conducted  by  Roberts  (1990).  Roberts  discussed  the  attributes  in 
organizations  that  were  successful  in  reducing  risks  associated  with  hazardous  operations. 
Her  label  for  this  type  of  organization  was  “high-reliability”  and  described  an 
organization  that  worked  in  a  hazardous  environment,  yet  produced  very  low  rates  of 
accidents.  The  CSA  study  was  based  on  a  model  of  high  reliability  organizations,  and 
produced  a  questionnaire  that  would  provide  feedback  to  the  HFQMB  concerning  the 
issues  of  command  climate,  morale  of  Naval  personnel,  workload,  resource  availability, 
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and  estimated  success  in  safety  intervention  programs.  This  newly  developed  tool,  the 
CSA,  could  be  used  by  the  Aviation  Safety  Officer  (ASO)  to  assess  the  organizations’ 
effectiveness  regarding  their  commands  safe  operation  (Nutwell  &  Sherman,  1997). 

3.  Maintenance  Climate  Assessment  Survey 

Safety  culture  within  an  organization  effects  the  entire  output  of  the  group.  Naval 
Aviation  is  recognized  as  a  high-risk  organization,  devastating  consequences  occur  if 
something  goes  wrong  within  any  aspect  of  the  organizations  operation.  Just  as  the 
concept  of  the  CSA  was  created  as  an  extension  of  the  principles  guiding  the  study  of 
high  reliability  organizations  (Ciavarelli  &  Figlock,  1997),  the  Maintenance  Climate 
Assessment  Survey  (MCAS)  became  the  maintenance  focused  extension  of  the  CSA 
(Baker,  1998).  The  MCAS  was  developed  with  the  notion  that  operational  climate  and 
maintenance  climate  hold  the  same  importance  within  the  unit,  therefore,  both  contribute 
to  safe  completion  of  the  organizations  mission.  With  maintenance  actions  contributing  to 
the  mishap  rate,  the  need  to  modify  a  survey  tool  to  address  the  issues  pertaining  to 
maintenance  became  relevant  (Baker,  1998). 

The  MCAS  has  a  focus  on  safety  climate  from  the  aviation  maintainer's 
viewpoint.  Baker’s  (1998)  goal  was  to  develop,  administer,  and  validate  a  questionnaire 
to  assess  the  effectiveness  of  Naval  Aviation  maintenance  operations  in  the  management 
of  risk.  Baker  concludes  that  the  developed  prototype  MCAS  with  a  few  noted 
exceptions  conformed  to  the  existing  Model  of  Organizational  Safety  Effectiveness 
(MOSE)  quite  readily,  he  successfully  adapted  the  model  to  make  it  maintenance  specific. 
Statistical  analysis  allowed  the  original  67  question  MCAS  to  be  reduced  to  a  more 
practical  length  of  35  questions.  Refinement  and  consideration  for  distribution  of  the 
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MCAS  offer  areas  of  further  development. 


C.  RESEARCH  OBJECTIVE 

The  objective  of  this  study  is  to  further  enhance  the  effectiveness  of  the  MCAS 
developed  by  Baker  at  the  Naval  Postgraduate  School  (Baker,  1998).  This  study  creates 
an  online  version  of  the  MCAS,  and  incorporates  this  version  of  the  MCAS  within  an 
electronic  distribution  system  and  tests  the  usability  of  the  online  system.  The  intent  is  to 
increase  the  availability  of  the  MCAS,  enhancing  present  efforts  to  reduce  maintenance 
related  mishaps  and  the  overall  mishap  rate. 

D.  STATEMENT  OF  PROBLEM 

Utilizing  the  distribution  capability  that  is  created  by  the  World  Wide  Web,  an 
online  version  of  the  MCAS  can  be  distributed  to  an  ever-increasing  audience  with 
virtually  no  increase  in  distribution  expense  (Lewis,  1998).  To  further  reduce  the 
maintenance  attributed  aviation  mishaps,  safety  process  and  programs  that  contribute  to 
safe  operation  will  need  to  improve.  Not  simply  seeking  improvements  in  process  and 
programs,  the  aviation  community  sought  the  invention  of  new  processes  to  enable  the 
desired  improvements  in  mishap  reduction  (Nutwell  &  Sherman,  1997). 

Distribution  of  information  on  the  World  Wide  Web  offers  the  opportunity  for  the 
Naval  Aviation  community  to  “invent”  the  new  process  that  it  desires,  bringing  about 
changes  in  safety  and  ultimately  reducing  the  mishap  rate.  As  Lewis  (1997)  states,  the 
World  Wide  Web  offers  increased  exposure  to  an  audience  without  increases  in 
distribution  costs,  and  as  a  distribution  medium,  the  World  Wide  Web  can  disseminate 
information  that  was  originally  designed  for  distribution  in  other  forms.  Specifically,  the 
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MCAS  was  created  to  be  distributed  in  paper  form,  with  modification  to  format,  the 
MCAS  can  be  reconfigured  to  World  Wide  Web  specifications.  Distribution  of  the 
developed  MCAS  by  an  effective  Web  based  method  will  enable  more  organizations  to 
use  this  tool  for  self-evaluation. 

The  main  thrust  of  this  study  focuses  on  the  effective  reconfiguration  and 
distribution  of  the  existing  MCAS  through  the  World  Wide  Web.  Usability  of  the  online 
version  of  the  MCAS  was  the  measurement  of  effectiveness.  Understanding  and 
measuring  the  effectiveness  of  the  safety  program  is  vital  to  the  reduction  of  human  factor 
caused  mishaps.  It  is  essential  to  provide  tools  to  the  aviation  organization  for  self- 
assessment  of  the  effectiveness  of  their  processes  and  programs.  Four  issues  are  relevant 
in  the  discussion  of  an  electronic  distribution  of  the  MCAS,  specifically,  this  thesis 
explores  the  following  questions: 

1.  Is  there  a  need  to  disseminate  and  administer  Web  based  materials  to  aviation 
units? 

2.  Do  U.S.  Navy  and  Marine  Corps  squadrons  have  the  necessary 
software/hardware  capabilities  to  utilize  the  World  Wide  Web? 

3.  Can  the  MCAS  be  effectively  distributed  and  administered  using  the  World 
Wide  Web? 

4.  Once  distributed,  do  personnel  have  the  necessary  ability  to  utilize  the 
electronic  version  of  the  MCAS? 

E.  SCOPE  AND  LIMITATIONS 

The  intent  of  this  study  is  to  gain  a  better  understanding  of  the  distribution 
capability  the  World  Wide  Web  offers  organizations  with  the  proper  hardware  and 
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software  in  place.  It  is  essential  to  determine  the  usability  of  an  online  version  of  the 
MCAS  for  use  by  Naval  Aviation  squadrons.  Subjects  from  the  Naval  Aviation  School 
of  Safety  will  be  used  due  to  their  close  proximity,  this  effort  represents  a  preliminary 
effort  to  measure  the  usability  of  the  online  version  of  the  MCAS.  An  exit  survey  will 
allow  the  users  to  rate  their  impression  of  this  method  of  MCAS  distribution. 
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II.  LITERATURE  REVIEW 


A.  OVERVIEW 

The  review  of  the  literature  considers  three  areas  pertinent  to  the  effective  online 
delivery  of  an  organizational  climate  assessment  tool.  Areas  covered  include: 

1)  Organizational  climate  and  its  assessment  to  provide  background  on  the  traditional 
process;  2)  High  Reliability  Organizations  (HRO)  as  a  model  for  effective  organizational 
climate  is  considered  and  its  application  to  the  military;  3)  Development  of  the  Command 
Safety  Assessment  (CSA)  and  the  Maintenance  Climate  Assessment  Survey  (MCAS)  for 
Naval  Aviation  based  on  the  HRO  model;  4)  The  software  development  process  entailed 
for  organizational  assessment  tool  delivery;  and  5)  The  Human  computer  interface 
concept,  functionality  and  usability.  The  information  is  summarized  and  considered  in 
context  of  the  research  questions. 

B.  ORGANIZATIONAL  CLIMATE  AND  ASSESSMENT 

Daft  (1989)  defines  organizational  climate  as: 

...the  set  of  key  values,  guiding  beliefs,  and  understandings  that  are  shared 
by  members  of  an  organization.  It  defines  basic  organizational  values,  and 
communicates  to  new  members  the  correct  way  to  think  and  act,  and  how 
things  ought  to  be  done.  It  represents  the  unwritten,  feeling  part  of  the 
organization,  (p.  503) 

Climate  is  the  unseen  part  of  an  organization  that  helps  members  make  sense  of 
organizational  events  and  activities  (Daft,  1989).  When  an  organization  enjoys 
widespread  agreement,  climate  within  the  organization  is  strong,  conversely,  if  little 
agreement  exists  within  the  organization,  climate  is  weak.  Daft  (1989)  also  notes  that 
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leadership  plays  a  large  role  in  the  organizational  climate.  He  states  that  simple  mandates 
of  policy  do  not  form  an  organizations  cultural  climate,  values  and  beliefs  come  from 
observing  leaders  within  the  organization.  Climate  is  an  important  feature  of  the 
organization,  and  if  a  change  in  organizational  climate  is  desired,  leadership  must 
implement  more  than  simple  policy  to  effect  the  needed  changes  (Schein,  1992). 

To  understand  more  fully  the  process  of  change,  it  is  helpful  to  study  the  needed 
elements  that  will  successfully  facilitate  it.  Burke  (1994)  looks  at  organizational 
development  as  it  relates  to  change.  Much  work  in  this  area  is  built  on  a  model 
developed  by  Lewin  (1958)  called  the  three-step  procedure  of  change.  The  first  step  of 
this  model  is  unfreezing  the  present  level  of  behavior,  which  is  no  more  than  reducing 
prejudice  and  clarifying  the  area  that  requires  change.  The  second  step  is  movement,  take 
action  that  will  change  the  social  system  from  its  original  state  of  behavior.  The  final  and 
third  step  is  refreeze,  or  establish  processes  that  will  secure  the  new  behavior  from 
change. 

Schein  (1992)  elaborates  on  Lewin’ s  model,  enlarging  and  defining  the  scope  of 
each  step.  He  further  sets  forth  useful  assumptions  when  studying  an  organization  and 
attempting  to  measure  culture.  Schein  notes,  it  is  important  to  keep  in  mind  that  the 
contextual  meaning  of  cultural  assumptions  can  only  be  fully  understood  by  the  members 
of  that  culture,  so  creating  a  vehicle  that  helps  the  members  of  the  group  understand  their 
own  culture  is  actually  more  important  than  the  researcher  gaining  this  perspective.  Since 
culture  is  a  shared  set  of  assumptions,  obtaining  initial  data  in  a  group  setting  is 
appropriate. 
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C.  HIGH  RELIABILITY  ORGANIZATION  (HRO) 

In  research  conducted  by  Roberts  (1990),  and  extended  by  Libuser  (1994),  an 
HRO  is  defined  as  an  organization  that  operates  in  a  hazardous  environment,  yet  produces 
very  low  rates  of  accidents  and  incidents.  HROs  seem  to  have  less  than  their  “fair  share” 
of  failures  despite  operating  in  a  complex,  demanding  environment  involving  significant 
risks  and  hazards.  Roberts  and  Libuser  note  that  these  HROs  had  certain  characteristics 
that  made  them  organizationally  effective  managers  of  risk.  Libuser  (1994)  went  on  to 
develop  a  Model  of  Organizational  Safety  Effectiveness  (MOSE)  that  categorized  five 
traits  that  were  common  among  this  type  of  organization,  these  traits  include:  process 
auditing,  reward  systems,  quality  of  operations,  risk  perception  and  command  control. 

HROs  are  found  in  many  areas  of  manufacturing  and  everyday  life  (Perrow, 
1984).  The  operation  of  power  plants,  oil  refineries  and  transportation  represent  a  few 
industries  where  accidents  are  rare  but  if  they  occur  have  catastrophic  and  far  reaching 
effects  (Reason,  1998).  The  operating  concepts  of  HROs  can  apply  to  many  other 
industries  where  accidents  can  not  be  tolerated,  the  food  industry,  drug  manufacturers, 
financial  institutions  are  just  a  few  of  the  organizations  that  characterize  groups  that 
depend  on  high  reliability.  Finally,  the  HRO  concept  can  also  be  extended  to  military 
applications  (Roberts,  1988). 

Reason  (1998)  describes  a  theory  of  active  and  latent  causes  of  accidents  and 
applies  these  causal  factors  to  HROs.  An  active  failure  is  one  that  is  directly  attributable 
to  an  action  or  violation  by  an  individual  directly  influencing  a  system.  Latent  conditions 
may  exist  for  years  within  a  system  and  not  cause  an  accident  until  the  proper  sequence  of 
events  come  together  simultaneously.  He  contends  the  perception  of  individuals  within 
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an  HRO  is  that  iron  clad  defenses  are  in  place  preventing  accidents,  however  in  practice, 
organizational  defenses  are  not  iron  clad,  and  may  have  certain  weaknesses.  When  the 
weaknesses  in  separate  defenses  occur  together  in  the  proper  sequence,  failure  may  result 
(See  Figure  3).  Meeting  peak  requirements  or  time  pressures  may  cause  individuals 
within  the  HRO  to  neglect  the  ongoing  process  of  finding  holes  in  the  accident  prevention 
strategy;  infrequent  mishap  occurrence  contributes  to  this  mindset  (Reason,  1998). 


Figure  3.  Perceived  and  Actual  Defenses  from  Risk 


Research  conducted  by  Roberts  (1990)  and  Libuser  (1994)  indicate  that  the 
complexities  of  systems  today  necessitate  new  strategies  for  prevention,  the  simple  direct 
causation  of  an  accident  by  one  individual  does  not  fit  a  HRO.  HROs  must  continually 
seek  new  methods  of  process  improvement  and  not  become  complacent  despite  the  fact 
they  suffer  from  few  accidents  (Perrow,  1984).  The  perception  that  these  organizations 
maintain  “iron  clad”  defenses  against  hazards  it  can  not  hamper  continual  process 
improvement  (Reason,  1998).  Strategies  to  achieve  a  HRO  would  include  a  strong 
hazard  reporting  system  with  no  fear  of  retribution  by  the  members,  as  well  as 
maintaining  a  flexible,  learning  culture  within  the  organization. 
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Bierly  and  Spender  (1995)  adapt  the  HRO  model  concept  for  military  application 
and  claim  that  a  nuclear  powered  submarine  fits  many  of  its  assumptions.  They  observe 
culture  interacts  with  the  formal  organization,  and  that  the  strong  organizational  culture 
within  the  nuclear  submarine  supports  the  formal  structure.  Roberts  (1988)  finds  an 
increasing  number  of  organizations  in  modem  society  fit  the  HRO  model,  observing  the 
size  and  complexity  of  the  aircraft  carrier  organization  coupled  with  a  seemingly  low 
incidence  history  in  dealing  with  high  risk  evolutions  qualified  it  as  a  HRO.  Baker  (1998) 
states,  “In  high-risk  endeavors  such  as  aircraft  operations,  it  is  essential  that  such  support 
be  flawless,  as  the  price  of  deviations  is  high.”  Naval  Aviation  as  a  whole  with  its 
complexity,  high  risk,  and  low  tolerance  for  error  fits  the  criteria  to  be  a  HRO. 

Building  on  the  work  of  Roberts  and  Libuser,  Ciavarelli  and  Figlock  (1997) 
tailored  the  MOSE  to  fit  the  conditions  found  in  Naval  Aviation.  They  created  the  CSA, 
a  tool  designed  to  assess  the  safety  climate  within  a  Naval  Aviation  squadron  from  an 
aircrew  perspective.  The  goal  of  this  research  was  to  develop  and  apply  a  methodology 
that  enabled  a  command  to  assess  it  performance  on  key  safety  issues,  providing  a 
feedback  tool  as  part  of  the  mishap  prevention  process.  Planned  follow  on  efforts  include 
cross-comparing  squadrons,  aircraft  type  and  demographic  variables  in  an  attempt  to 
construct  an  empirical  foundation  for  assessing  hazards  and  risk. 

Baker  (1998)  furthered  the  efforts  of  the  CSA  by  designing  and  validating  the 
MCAS  that  would  accomplish  the  same  objectives  as  the  CSA,  but  from  the  perspective 
of  the  aviation  maintainer.  With  the  help  of  maintenance  experts,  he  took  a  list  of 
maintenance  topics  and  created  67  questions  specifically  tailored  to  aviation  maintenance. 
Baker’s  initial  67  question  survey  through  factor  analysis  was  further  refined  and  reduced 
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to  a  35  question  instrument.  The  MCAS  was  successfully  proven  to  support  each  of  the 
MOSE  components. 

Currently,  Naval  Postgraduate  School  Operations  Research  Department  students 
are  engaged  in  taking  the  MCAS  and  validating  it  by  analyzing  the  results  within  an 
aircraft  community,  and  across  aircraft  communities  (B.  Goodrum,  Personal 
Communication.  1999;  T.  Oneto,  Personal  Communication.  1999).  To  further  enhance 
the  effectiveness  of  the  MCAS,  this  study  seeks  to  provide  a  wide  reaching  delivery 
system  on  the  World  Wide  Web.  Effective  distribution  will  involve  the  creation  of  an 
electronic  delivery  system,  and  the  usability  testing  of  that  system. 

D.  SOFTWARE  TOOLS 

The  software  necessary  to  carry  out  the  successful  development  and 
administration  of  the  online  MCAS  includes  two  categories:  developmental  software,  and 
user  software  (Alexander  &  Tate,  1999).  The  developmental  software  in  this  research 
created  both  the  Web  site,  and  also  the  online  version  of  the  MCAS.  The  user  software  is 
the  tool  each  user  must  employ  to  utilize  the  online  MCAS  once  developed;  this  software 
is  determined  by  the  end  user  and  is  a  World  Wide  Web  browser.  Both  categories  of 
software  and  its  employment  are  discussed  below. 

1.  Developmental  Software 

Perseus  Survey  Solutions  for  the  Web  is  the  software  that  was  used  for  the 
transformation  of  the  MCAS  to  an  online  version  of  the  same.  The  Perseus  Corporation 
makes  this  software  product  Microsoft  Office  compatible  (Strebe  &  Perkins,  1998). 
Compatibility  with  Microsoft  products  allows  for  future  expansion  of  the  online  MCAS 
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to  include  functions  contained  in  Microsoft  Access  and  Microsoft  Excel. 

The  most  common  application  of  this  development  software  is  construction  and 
distribution  of  surveys,  but  its  other  functionality  includes  the  development  of  inquiry 
forms,  feedback  questionnaires  and  order  forms  (Perseus  Survey  Solutions  User  Guide, 
1998).  Functioning  as  a  word  processor.  Survey  Solutions  for  the  Web  acts  as  an 
interface  between  the  plain  language  text  formulation  and  translation  to  Hypertext 
Markup  Language  (HTML).  Once  textual  data  is  converted  to  HTML,  it  may  be  posted 
on  a  server  and  users  may  access  the  stored  data  through  their  computer  and  an  Internet 
browser.  The  storage  and  availability  of  these  files  is  further  described  in  the  discussion 
of  the  World  Wide  Web. 

Data  collection  is  the  final  function  that  the  Perseus  software  package  carries  out 
as  part  of  the  design  of  the  online  MCAS.  Once  collected  through  the  user  response  to 
questions  within  the  online  MCAS,  data  is  packaged  in  a  standard  tab  separated  form 
(ASCII  TSV)  which  may  be  further  stored,  or  interpreted  using  Microsoft  Access  or  any 
program  that  allows  the  ASCII  TSV  data  entry  (Perseus  Survey  Solutions  User  Guide, 
1998). 

The  second  software  development  tool  used  was  Netscape  Navigator  Gold 
Version  4.5.  This  software  package  was  used  in  the  development  of  the  Web  site  that 
presented  the  online  MCAS  to  the  user.  Netscape  Navigator  Gold  utilizes  a  powerful 
What- You-See-Is- What- You-Get  (WYSIWYG)  HTML  generator  that  uses  many  of  the 
same  functions  found  in  word  processors  (Authoring  Guide,  Netscape  Navigator  Gold, 
1995).  Maintenance  Aviation  Safety  material  was  gathered  from  existing  sources  and 
combined  with  original  material  to  form  a  distribution  vehicle  for  the  MCAS. 
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2.  User  Software 


The  final  piece  of  software  used  in  this  project  is  the  Internet  Browser  (browser). 
A  browser  is  defined  as  “a  computer  program  that  enables  the  user  to  read  HYPER-TEXT 
from  disk  files  or  on  the  World  Wide  Web”  (Downing,  Covington  &  Covington,  1996). 
A  browser  is  the  software  each  end  user  must  utilize  to  access  the  online  MCAS,  and 
becomes  the  most  dynamic  portion  of  the  all  the  software  packages  used.  The  dynamic 
nature  of  Internet  Browsers  is  discussed  more  fully  in  subsequent  sections  and  will  be  an 
ongoing  concern  in  a  Web  based  project. 

E.  WORLD  WIDE  WEB 

1.  Overview  of  the  World  Wide  Web 

The  World  Wide  Web  is  a  unified,  interconnected  interface  to  the  vast  amount  of 
information  stored  on  the  computers  around  the  world  (Sobell,  1995)'  Information  can  be 
displayed  on  the  World  Wide  Web  using  a  combination  of  text,  graphics,  animations  and 
audio  to  form  dynamic  documents.  Once  created,  these  documents  are  displayed  on  a 
user  computer  by  an  Internet  browser.  A  browser  provides  easy  access  to  Web  sites  that 
are  posted  on  many  servers  connected  to  the  World  Wide  Web.  A  series  of  Clients, 
computers  that  receive  information  from  other  computers,  and  Servers,  a  computer  that 
provides  services  to  another  computer,  make  up  the  Internet. 

Composed  of  HTML  code,  a  Web  site  is  made  up  of  one  or  more  pages  to  create 
an  organized  presentation  of  material  through  the  use  of  Hyperlinks.  The  Authoring 
Guide,  Netscape  Navigator  Gold  (1995)  defines  the  Web  site  as  a  package  that  contains; 
topics  that  can  be  linked  to  from  other  Web  sites,  may  require  the  use  to  scroll  the  browse  1 
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window  to  see  the  entire  page,  can  contain  links  to  graphic  files  and  other  pages,  and 
finally  can  be  located  on  a  server  so  that  other  users  may  access  the  page.  Below,  Figure  2 
illustrates  the  composition  of  a  Web  site  that  is  made  up  of  several  Web  pages,  connected 
by  use  of  Hyperlinks. 

Hyperlinks  are  the  mechanism  that  allow  an  entire  Web  site  composed  of  many 
Web  pages  to  be  connected  in  a  logical  sequence.  The  use  of  Hyperlinks  to  tie  together 
multiple  Web  pages  of  the  same  theme  allow  a  user  to  navigate  a  rather  complex  Web  site 
with  ease  and  little  confusion.  The  World  Wide  Web  offers  and  opportunity  to  distribute 
information  to  a  vast  audience  at  very  little  cost  (Lewis,  1998). 


Web  pages  connected 
through  Hyperlinks 


Web  page 


Figure  4.  Overview  of  a  Web  site 


WebSite 


2.  The  Internet  Browser 

The  browser  is  an  integral  part  of  using  any  online  software  applications, 


including  the  online  MCAS.  Each  browser  interprets  HTML  code  by  applying  a  set  of 
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defaults  established  by  the  individual  user.  A  default  is  a  set  of  assumptions  that  a 
browser  makes  unless  it  is  given  instructions  to  the  contrary  (Downing,  Covington  & 
Covington,  1996).  These  defaults  within  the  browser  determine  how  the  browser 
interprets  the  HTML  code  and  displays  the  information  on  the  users  monitor.  Different 
interpretations  of  HTML  code  result  in  different  views  of  the  same  information.  For  this 
reason,  two  browsers  may  interpret  the  same  HTML  code  differently.  In  the  design  of 
Web  sites,  it  becomes  necessary  to  consider  the  different  default  settings  users  will  utilize 
and  anticipate  the  effects  this  may  have  on  data  within  a  Web  site  application  (Randall, 
1999). 

3.  The  Client/Server  Model 

The  Client/Server  model  is  the  basic  model  that  enables  individual  users  to  access 
a  Web  site  at  any  given  time  (Sobell,  1995).  The  Server  is  a  program  that  runs  all  the 
time  waiting  to  receive  a  request  for  the  information  it  contains  in  its  data  stores  or 
memory.  World  Wide  Web  clients  are  the  computer  users  who  access  a  server  through 
the  use  of  a  browser.  The  Client/Server  interaction  occurs  using  a  computer  language 
known  as  Hypertext  Transfer  Protocol  (HTTP).  HTTP  is  a  computer  language  that  works 
across  all  platforms;  consequently,  whether  the  Client  and  Server  are  the  same  type  of 
computer  does  not  inhibit  the  ability  for  the  two  computers  to  communicate  (Downing, 
Covington  &  Covington,  1996).  When  the  Client  requests  information  from  the  Server 
via  HTTP,  the  request  is  acknowledged  and  data  is  sent  back  to  the  Client.  (See  Figure  5) 

The  Client/Server  Model  is  responsible  for  enabling  the  efficiencies  in 
distribution  of  online  material  throughout  the  World  Wide  Web  (Tanenbaum,  1992). 
Once  posted  on  a  Server,  the  Online  MCAS  will  incur  the  same  distribution  cost  as  it 
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reaches  an  ever-increasing  audience.  The  efficiency  in  online  distribution  is  analogous  to 
the  infinite  shelf  space  retailer  of  the  ffictionless  economy  (Lewis,  1997).  The  ability  to 
distribute  training  materials  at  a  small,  fixed  cost,  to  an  audience  that  is  potentially 
infinite  represents  a  method  of  improvement  for  training  that  may  facilitate  reaching 
mishap  reduction  goals  set  within  the  organization  of  Naval  Aviation. 


Client 


Figure  5.  Illustration  of  the  Client/Server  Model 


F.  HUMAN  USABILITY  ISSUES 

Hawkins  (1998)  observes  that  “Human  factors  is  about  people... It  is  about  their 
relationship  with  machines  and  equipment,  with  procedures  and  with  the  environment 
about  them.”  He  illustrates  this  relationship  through  the  SHEL  model,  where  the  elements 
of  Software  (e.g.,  training,  procedures,  etc.),  Hardware  (e.g.,  controls,  displays,  etc.), 
Environment  (e.g.,  temperature,  lighting,  etc.),  and  Liveware  (e.g.,  operator,  systems, 
etc.),  pair  to  form  a  human  system  interface.  Liveware  is  considered  the  hub  of  the  model 
and  all  remaining  components  must  be  effectively  matched  to  this  central  component. 
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Human  factors  principles  have  traditionally  been  applied  to  hardware  devices  such  as 
monitors,  keyboards,  etc.  as  well  as  workstation  design  to  optimize  their  fit  with  human 
needs  and  capabilities  (Grandjean,  1987).  The  proliferation  of  hardware  and  software 
technology  necessitated  the  same  intensive  application  of  human  factors  technology  to 
enhance  software  Human  Computer  Interface  (HCI)  design  (Brown,  1989).  Today,  most 
software  and  design  applications  integrate  HCI  design  concepts  including:  fonts,  color, 
etc.,  as  well  as  screen  layout  and  functionality. 

The  “browsers”  that  the  World  Wide  Web  is  accessed  through  have  gone  passed 
many  HCI  design  iterations  to  maximize  their  efficiency  and  ease  of  use  (Frederickson- 
Mele,  Levi,  &  Conrad,  1999).  Web  software  and  that  used  in  Web  site  development  have 
also  been  crafted  using  HCI  concepts  with  similar  objectives  as  in  software  development. 
Using  the  Internet  as  a  distribution  method  raises  unique  design  issues  that  are  dependent 
on  user  terminals  and  pre-selected  defaults,  consequently.  World  Wide  Web  applications 
benefit  from  the  HCI  research  in  the  browser  they  are  viewed  through.  This  unique  set  of 
user  circumstances  requires  that  Web  sites  be  tested  for  usability  by  their  intended 
audience. 

1.  Usability  Testing 

Software  and  its  design  presents  many  challenges  to  the  user  in  trying  to 
accomplish  a  given  task.  A  reasonable  goal  of  any  software  project  should  be  the  ease  of 
use  of  the  final  product  (Jordan,  1998).  Usability  issues  are  not  only  important  within  the 
realm  of  consumer  products,  but  also  those  products  used  within  a  commercial  or 
professional  context  (Jordan,  1998).  Past  studies  indicate,  with  the  exception  of  safety 
testing,  usability  testing  takes  place  in  the  worst  possible  arena,  the  marketplace  (Randall, 
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1999).  In  other  words,  Randall  (1998),  states  whether  a  product  was  usable  or  not  was 
left  up  to  the  end  user,  after  the  purchase  or  attempted  use  of  the  final  product.  Formal 
usability  testing  became  recognized  about  a  dozen  years  ago  in  the  computer  software 
industry.  Usability  testing  does  not  speed  the  design  process  up,  but  if  user  satisfaction  is 
the  goal,  the  increased  development  time  is  well  spent.  Constructing  a  site  to  meet  the 
needs  of  your  intended  audience  increases  the  impact  the  overall  presentation  has  upon 
the  user  (Frederickson-Mele,  Levi,  &  Conrad,  1999). 

Delivering  a  prototype  of  the  product  to  a  test  group  followed  by  a  questionnaire 
is  a  method  Jordan  (1998)  found  beneficial  in  determining  the  attitudes  and  expectations 
of  potential  product  users.  Jordan  specifies  that  data  obtained  from  usability  studies 
should  answer  the  following  questions: 

•  Are  there  usability  faults  in  the  product?  If  so,  what  are  these  faults? 

•  How  can  these  usability  faults  be  solved? 

•  What  are  user  attitudes  towards  the  use  of  this  product? 

•  What  design  changes  are  required  to  make  the  users’  attitudes  more  positive? 

Open  ended  questions,  allow  the  respondent  to  write  their  own  answers  to  questions  and 
prove  particularly  useful  in  determining  what  the  important  issues  were  to  the  users.  A 
usability  study  enhances  the  effectiveness  of  modification  to  a  prototype,  ultimately;  the 
final  product  will  have  more  utility  to  the  end  user  (Jordan,  1998). 

2.  Writing  For  the  Web 

When  writing  for  Web  sites,  studies  have  shown  that  users  do  not  actually  read: 
instead,  they  scan  the  text  (Nielsen,  1994).  This  finding  suggests  a  style  of  writing  that 
can  be  absorbed  through  scanning  rather  than  through  in-depth  study.  During  Nielsen’s 
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four  years  of  usability  testing,  he  considered  site  architecture,  navigation,  search,  page 
design,  and  style  yielding  surprising  results.  When  asked  to  comment  on  the  Web  site, 
users  critiqued  content  quality  and  relevance  to  a  much  greater  extent  than  navigational 
issues  or  page  elements  (Nielsen  &  Sano  1994;  Nielsen,  1997).  This  suggested  content 
would  play  a  very  large  part  in  the  users  perception  of  the  Web  site.  Morkes  and  Nielsen 
(1997)  suggest  three  main  content  oriented  conclusions  be  drawn  from  their  study. 

1 .  Users  do  not  read  on  the  Web;  instead  they  scan  the  pages,  trying  to  pick  out  a  few 
sentences  or  even  parts  of  sentences  to  get  the  information  they  want. 

2.  Users  do  not  like  long,  scrolling  pages:  they  prefer  the  text  to  be  short  and  to  the 
point. 

3.  Users  detest  anything  that  seems  like  marketing  fluff  or  overly  hyped  language 
and  prefer  factual  information. 

It  is  clear  that  users  require  concise  presentations  of  content  when  on  the  Web,  and  have 
very  low  tolerance  for  superfluous  words  or  ideas  in  the  material. 

Morkes  and  Nielsen  (1997)  state  that  writing  effectively  for  the  Web  is  a  function 
of  sensitivity  to  the  users  needs  balanced  with  the  message  you  wish  to  covey  to  the  user. 
Writing  in  a  concise,  scannable  style  would  seem  to  fit  the  users  need  for  quick  fact 
finding,  while  ensuring  content  was  available  to  the  user.  It  was  noted  that  when  a  page 
comes  up,  the  readers  attention  shifts  to  center  screen  and  the  body  of  the  text  was  read 
before  they  looked  over  headers  or  other  navigational  elements  that  made  up  the  page 
(Morkes  &  Nielsen,  1997).  Similar  findings  on  the  importance  of  content  resulted  from 
the  work  of  other  similar  studies  of  online  materials  (Spool,  Scanlon,  Schroeder,  Snyder, 
&  DeAngelo,  1997).  This  indicates  the  need  to  maintain  a  concise  writing  style  on  the 
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Web  page,  paying  particular  attention  to  the  relevance  of  the  material  presented  to  the 
intended  user. 

G.  SUMMARY 

An  understanding  of  organizational  climate  leads  research  in  the  direction  of 
categorizing  Naval  Aviation  as  a  HRO.  As  a  HRO,  Naval  Aviation  safety  programs  and 
processes  can  be  analyzed  and  improved  using  the  CSA  and  MCAS.  Working  within  the 
confines  of  current  distribution  methods,  the  WWW  offers  unique  opportunities  for 
widespread,  inexpensive  distribution  of  safety  material.  Special  consideration  must  be 
given  to  design  issues  regarding  usability  of  software  applications.  A  vehicle  to  distribute 
the  online  version  of  the  MCAS  will  further  the  need  to  assess  safety  program 
effectiveness  in  Naval  Aviation. 
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III.  METHODS 


A.  RESEARCH  APPROACH 

Naval  Aviation  seeks  methods  to  improve  its  process  and  programs  regarding 
safety  in  an  effort  to  reduce  mishap  rates.  The  MCAS  developed  by  Baker  (1998)  was 
designed  to  be  utilized  by  squadrons  to  assess  the  effectiveness  of  their  safety  programs 
from  a  maintainers  perspective,  making  it  possible  to  measure  the  safety  climate  within 
an  aviation  command.  Making  the  MCAS  available  to  more  aviation  units  through 
effective  distribution  on  the  World  Wide  Web  is  the  goal  of  this  study. 

To  measure  the  effectiveness  of  the  online  MCAS,  a  prototype  Web  site  was 
developed  and  distributed  to  a  sample  user  group.  Each  subject  tested  the  Naval  Aviation 
Maintenance  Safety  Web  site  with  a  prepared  task  list  that  required  them  to  navigate 
through  the  pages  of  the  Web  site.  At  the  completion  of  the  task  list,  the  subject  would 
have  viewed  all  the  pages  of  the  Web  site,  and  presumably  formed  an  opinion  on  its 
effectiveness.  Subjects  upon  completing  the  task  list  filled  out  an  exit  survey  composed 
of  demographic  questions  of  their  computer  background  and  perusal  of  the  prototype  Web 
site. 

B.  DATA  COLLECTION 

1.  Subjects 

Students  attending  the  Aviation  Safety  Officer  Course  (ASO)  at  the  School  of 
Aviation  Safety  served  as  subjects  in  the  study.  Selection  to  the  ASO  course  is  made  by 
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individual  Aviation  Commands  and  yields  a  cross  section  of  the  Naval  Aviators  and 
Naval  Flight  Officers  from  all  aircraft  communities  within  the  U.S.  Navy  and  Marine 
Corps.  Squadron  ASOs  are  responsible  for  the  safety  programs  in  all  squadrons  and  will 
likely  be  one  of  the  primary  users.  Each  subject  from  this  test  group  will  have  a  direct 
influence  on  the  safety  climate  within  their  squadrons. 

2.  Apparatus 

The  Naval  Postgraduate  School’s  School  of  Aviation  Safety  provides  a  computer 
lab  for  the  ASO  students  during  their  training.  Each  Officer  is  issued  a  password  that 
allows  computer  usage  through  a  group  account.  The  computers  in  the  lab  each  had  a  full 
functioning  prototype  of  the  Aviation  Maintenance  Safety  Web  Site.  After  a  subject 
gained  access  to  the  computer  through  use  of  the  group  account,  the  icon  “Aviation 
Maintenance  Safety  Web  site”  resided  on  the  computer  desktop.  Once  the  Web  site  was 
opened,  it  appeared  as  any  World  Wide  Web  site,  although  this  prototype  was  installed  on 
the  hard  drive  of  the  computer  rather  than  on  a  server  on  the  Internet.  The  prototype  was 
kept  locally  on  the  lab  computers  to  facilitate  changes  that  would  occur  in  its 
development.  Microsoft  Internet  Explorer  was  the  user  software  provided  to  the  subjects. 
All  of  the  functions  within  Internet  Explorer  operated  in  the  same  manner  as  if  the 
application  was  taken  from  the  World  Wide  Web. 

3.  Instrument 

The  exit  survey  consisted  of  three  parts:  1)  Demographic  items  or  computer 
experience,  2)  Likert  type  questions  assessing  feeling  towards  the  Web  site,  and  3)  Open- 
ended  items  to  elicit  subjective  opinions.  The  first  part,  collecting  demographic 
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information,  allowed  the  subject  to  choose  from  a  list  of  descriptors  (e.g.  Squadron  level, 
Intermediate  level,  etc.).  The  second  and  third  part  of  the  survey  was  a  combination  of 
Likert  style  questions  and  open-ended  questions  that  allowed  the  subjects  to  inteiject 
comments  they  felt  were  relevant.  The  Likert  questions  were  constructed  using  a  five 
point  rating  scale  with  verbal  anchors,  as  follows:  Strongly  Agree,  Agree,  Neutral, 
Disagree,  and  Strongly  Disagree.  The  open-ended  questions  were  included  in  the 
questionnaire  to  elicit  respondent  viewpoints  regarding  such  factors  as  the  overall 
impression  of  the  Web  based  distribution,  and  comments  regarding  improving  the 
usability  of  the  Web  site.  The  open-ended  questions  provided  subjects  an  opportunity  to 
comment  on  aspects  of  the  prototype  Web  site  they  felt  were  not  adequately  covered  in 
the  previous  portions  of  the  survey  (See  Appendix  D  for  survey). 

4.  Procedure 

The  subjects  were  divided  into  two  groups  of  26  and  presented  with  an  overview 
presentation  of  the  study.  The  overview  consisted  of  a  projected  computer  demonstration 
of  the  Aviation  Maintenance  Safety  Web  site,  and  distribution  of  the  materials  necessary 
to  carry  out  the  user  test.  The  materials  used  to  direct  the  subjects  exposure  to  the  Web 
site  included:  Instructions  for  viewing  the  Aviation  Maintenance  Web  site  (Appendix  B); 
Aviation  Maintenance  Safety  Web  Site  Task  List  (Appendix  C);  and  User’s  Impression  of 
the  Maintenance  Safety  Web  Site  and  MCAS  (Appendix  D).  The  Instructions  for 
viewing  the  Aviation  Maintenance  Web  Site  was  a  guide  for  the  subject  to  follow 
enabling  him/her  to  log  on  to  the  group  account  provided  in  the  computer  lab  and  access 
the  Web  site  installed  on  the  computer.  As  a  portion  of  the  group  overview,  specific 
questions  subjects  expressed  on  using  the  computers  in  the  lab  were  addressed.  The  Task 
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list  was  designed  to  take  the  subject  through  a  series  of  planned  navigation  routes  within 
the  Web  site.  By  completing  the  task  list,  the  subject  would  have  navigated  through  the 
entire  prototype  of  the  Aviation  Maintenance  Safety  Web  site. 

C.  DATA  TABULATION 

The  data  was  transcribed  from  the  exit  survey  into  the  software  program  SPSS 
(SPSS  v.  9.0).  Multiple  choice  responses  were  each  assigned  a  value  from  one  through 
four.  The  Likert  style  questions  were  based  on  a  five  point  scale,  these  questions  were 
coded  into  the  SPSS  software  using  five  (5)  through  one  (1)  corresponding  to  the 
respective  anchors  (Strongly  Agree,  Agree,  Neutral,  Disagree,  and  Strongly  Disagree). 
Once  the  data  was  transcribed,  the  analysis  tools  in  the  software  SPSS  were  used  to 
generate  descriptive  statistics  including  the  mean,  standard  deviation,  range,  and 
frequency  distribution.  Content  analysis  was  conducted  on  the  responses  provided  from 
the  open-ended  survey  questions. 

D.  DATA  ANALYSIS 

1.  Analysis  Strategy 

The  data  from  the  surveys  was  transferred  to  the  statistical  analysis  program  SPSS 
(SPSS  v.9.0)  for  analysis  and  interpretation.  One-way  Analysis  of  Variance  (ANOVA) 
testing  matched  the  level  of  user  experience.  Independent  Variable  (IV),  with  selected 
Dependent  Variables  (DV)  gathered  in  the  survey.  The  one-way  ANOVA  was  used  to 
assist  in  answering  the  difference  (group  comparison)  issues  raised  from  the  study. 
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IV.  RESULTS 


A.  FREQUENCY  AND  CONTENT  ANALYSIS 

The  21  item  exit  survey  was  administered  to  52  subjects  with  a  response  rate  of 
88%  (n  =  46).  The  test  users  represented  a  cross  section  of  the  aviation  commands  that 
make  up  the  squadrons  of  the  Fleet.  Each  aviation  command  is  required  to  have  an 
officer  trained  by  the  School  of  Aviation  Safety;  as  a  result,  the  entire  fleet  is  represented 
in  this  school  at  any  given  time.  The  sample  included  flight  qualified  Officers  from  Naval 
and  Marine  Corps  Aviation  and  did  not  include  Non-Commissioned  Officers  or  Enlisted 
personnel. 

1.  Demographic  Information 

The  material  collected  in  Part  I,  the  demographic  portion  of  the  exit  survey 
determined  the  experience  level  each  subject  had  with  computers,  as  well  as,  the  Internet 
and  Internet  browsers.  This  information  is  later  used  to  determine  if  experience  level 
with  Internet  browsers  affected  a  subject’s  level  of  satisfaction,  and  impacted  the  usability 
of  the  Web  site.  Question  one  on  the  survey  revealed  that  all  participants  were  members 
of  aviation  units  that  performed  maintenance  at  the  squadron  level  (n  =  46,  100%). 
Question  two  asked,  “How  long  have  you  been  using  an  Internet  browser?”  Table  1 
shows  the  breakdown  regarding  experience  each  subject  had  with  Internet  browsers.  A 
total  of  82  percent  (n  =  38)  of  the  subjects  had  at  least  one  year  of  experience  using  the 
Internet. 
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Table  1.  User  Experience  With  a  Computer  Browser 


2.  User  Satisfaction  With  the  Web  Site 

Part  II  of  the  exit  survey  examined  the  subject’s  impressions  of  the  layout  and 
content  of  the  Web  site.  Questions  6  through  9  specifically  targeted  the  subjects’  overall 
impression  of  the  Web  site.  Question  6  asked,  “My  tour  of  the  Web  site  was  very 
interesting.”  and  the  mean  was  3.87  (SD=.686).  The  range  for  question  6  was  4,  and  the 
histogram  of  the  frequency  distribution  is  presented  in  Figure  6.  Clearly  many 
respondents  (74%)  agreed  that  the  tour  of  the  Web  site  was  interesting,  with  only  one 
person  disagreeing  with  this  statement. 


Strongly  Agree  Neutral  Disagree  Strongly 
Agree  Disagree 


Figure  6.  “Interesting” 
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Question  7  stated,  “I  found  the  Web  site  easy  to  navigate”  and  the  mean  was  4.35, 
(SD  =  .822).  The  range  for  question  7  was  4,  and  the  histogram  of  the  frequency 
distribution  is  presented  in  Figure  7.  The  vast  majority  (91%)  agreed  with  this  statement, 
while  three  persons  disagreed. 


Agree  Disagree 


Figure  7.  “Easy  to  Navigate” 


Question  8  asked,  “I  feel  the  information  on  this  Web  site  was  in  a  logical  form.” 
and  the  mean  was  4.16,  (SD  =  .556).  The  range  for  question  8  was  3,  and  the  histogram 
of  the  frequency  distribution  is  presented  in  Figure  8.  Most  respondents  (91%)  agreed 
that  information  was  logically  represented,  with  only  four  persons  neutral  on  this  topic. 
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Figure  8.  “Web  Site  in  Logical  Form” 


Question  9  asked,  “The  information  presented  in  the  Web  site  is  relevant  to 
maintenance  safety”  and  the  mean  was  4.37  (SD  =  .488).  The  range  for  question  9  was  2, 
and  the  histogram  of  the  frequency  distribution  is  presented  in  Figure  9.  All  subjects 
(100%)  either  strongly  agreed  or  agreed  with  this  statement. 
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Figure  9.  “Information  Relevant” 


After  completing  the  Likert  style  questions  in  the  first  section  of  Part  D,  the 


respondent  was  asked  to  comment  on  his/her  impression  of  the  site  through  the  open 
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ended  question,  “My  overall  impression  of  the  Web  site  is:”  Below  are  listed  some  of 
comments  that  typified  the  response  to  this  question. 

“Looks  like  it’ll  be  useful.  Especially  the  MCAS  and  MCAS  analysis  tools.” 

“Simple,  straight  forward  and  should  have  useful  information  for  safety  issues.” 

“Very  user  friendly  Web  site  which  contained  both  interesting  and  useful 

information.” 

“Should  be  a  good  tool  for  promoting  safety  and  existing  tools.” 

Subjects  expressed  general  enthusiasm  for  the  Web  site  and  thought  it  would  be  useful  in 
the  distribution  of  safety  related  material. 

The  next  four  questions  (10-13)  of  the  exit  survey  looked  at  consistency  issues. 
Consistency  is  important  in  both  term  usage  and  implicit  instructions,  this  makes 
navigation  through  the  Web  site  possible  with  .minimum  confusion.  Whether  consistency 
is  achieved  was  determined  by  the  impression  of  the  respondent.  These  four  Likert  style 
questions  were  posed  to  allow  each  user  of  the  prototype  to  comment  on  their  perceived 
consistency  throughout  the  exercise. 

Question  10  asked,  “During  my  tour  of  the  Web  site,  I  found  the  use  of  terms 
consistent”  and  the  mean  was  4.05  (SD  =  .630).  The  range  to  question  10  was  4,  and  the 
histogram  of  the  frequency  distribution  is  presented  in  Figure  10.  The  majority  of  users 
(74%)  agreed  with  this  statement,  with  only  two  users  neutral  and  two  disagreeing. 


37 


Strongly  Agree  Neutral  Disagree  Strongly 
Agree  Disagree 


Figure  10.  “Use  of  Terms  Consistent” 


Question  1 1  continued  with  consistency  by  asking,  “During  my  tour  of  the  Web 
site,  I  found  The  position  of  screen  instructions  were  consistent”  and  the  mean  of  this 
question  was  4.31  (SD  =  .510).  The  range  to  question  1 1  was  3,  and  the  histogram  of  the 
frequency  distribution  is  presented  in  Figure  11.  The  majority  (98%)  agreed  with  this 
statement,  and  only  one  user  was  neutral. 


Strongly  Agree  Neutral  Disagree  Strongly 
Agree  Disagree 


Figure  1 1 .  “Screen  Instructions  Consistent” 
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Question  12  asked,  “I  found  the  results  obtained  from  the  Web  site  were 
predictable”  and  the  mean  was  3.94  (SD  =  .611).  The  range  of  question  12  was  3,  and  the 
histogram  of  the  frequency  distribution  is  presented  in  Figure  12.  A  majority  of  the  users 
(78%)  found  that  they  received  consistent  results  from  the  Web  site,  although,  nearly  on 
quarter  of  the  respondents  (22%)  were  neutral  on  this  aspect. 


Strongly  Agree  Neutral  Disagree  Strongly 
Agree  Disagree 


Figure  12.  “Results  Predictable” 

Question  13  asked,  “During  my  tour  of  the  Web  site,  I  found  adequate  instructions 
were  provided  to  navigate”  and  the  mean  was'4.22  (SD  =  .757).  The  range  for  question  13 
was  4,  and  the  histogram  of  the  frequency  distribution  is  presented  in  Figure  13.  Most 
users  (89%)  aggreed  adequate  instructions  were  provided,  while  three  persons  were 
neutral  and  two  persons  disagreed. 
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Strongly  Agree  Neutral  Disagree  Strongly 
Agree  Disagree 


Figure  13.  “Adequate  Navigation  Instructions” 

The  next  five  questions  on  the  survey  (14-18)  addressed  the  organization  and 
clarity  of  the  information  presented  in  the  prototype  Web  site.  These  questions 
concentrate  on  the  experience  of  the  user  while  learning  the  methods  employed  to 
navigate  the  site.  An  open-ended  question  at  the  end  of  this  section  provided  the  subject 
an  opportunity  to  voice  opinions  not  covered  in  the  Likert  scale  questions. 

Question  14  asked,  “I  feel  that  the  information  on  the  Web  site  is  easy  to 
understand”  and  the  mean  was  4.16  (SD  =  .868).  The  range  for  question  14  was  4,  and  the 
histogram  of  the  frequency  distribution  is  presented  in  Figure  14.  While  a  majority 
agreed  (87%)  the  information  was  easy  to  understand,  two  subjects  were  neutral,  with  two 
disagreeing. 
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Strongly  Agree  Neutral  Disagree  Strongly 
Agree  Disagree 


Figure  14.  “Information  Easy  to  Understand” 

Question  1 5  stated,  “I  feel  that  it  was  easy  to  learn  to  use  this  Web  site”  and  the 

mean  was  4.42  (SD  =  .652).  The  range  for  question  15  was  3,  and  the  histogram  of  the 
frequency  distribution  is  presented  in  Figure  15.  A  large  majority  (91%)  agreed  with  this 
statement,  while  half  (50  %)  strongly  agreed  with  the  ease  of  learning  the  Web  site. 


Strongly  Agree  Neutral  Disagree  Strongly 
Agree  Disagree 


Figure  15.  “Ease  of  Learning  the  Web  Site” 
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Question  16  stated,  “I  feel  that  the  organization  of  material  on  this  Web  site  is 
clear”  and  the  mean  was  4.05  (SD  =  .815).  The  range  for  question  16  was  4,  and  the 
histogram  of  the  frequency  distribution  is  presented  in  Figure  16.  Most  users  (82%)  the 
organization  was  clear,  while  five  users  were  neutral  and  three  users  disagreed. 
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Agree  Disagree 

Figure  16.  “Organization  of  Material” 


Question  17  stated,  “I  feel  that  I  could  effectively  use  the  information  on  this  Web 
site”  and  the  mean  was  4.35  (SD  =  .566).  The  range  for  question  17  was  3,  and  the 
histogram  of  the  frequency  distribution  is  presented  in  Figure  17.  The  majority  of 
respondents  (96%)  agreed,  with  two  persons  being  neutral. 
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Agree  Disagree 


Figure  17.  “Effectively  Use  the  Information” 


Question  18  stated,  “I  feel  that  when  fully  implemented,  this  Web  site  will  have 
all  the  functions  and  capabilities  I  would  expect  it  to  have”  and  the  mean  was  4  (SD  = 
.760).  The  range  for  question  18  was  4,  and  the  histogram  of  the  frequency  distribution  is 
presented  in  Figure  19.  Most  users  (81%)  agreed,  whereas  nine  users  disagreed. 
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Figure  18.  “Web  Site  has  all  Functions  I  Expect” 


The  open  ended  question  in  the  final  section  of  Part  II  of  the  Exit  Survey  asked, 
“To  improve  the  usability  of  this  Web  site,  I  would.”  The  purpose  of  this  question  was 
twofold;  first,  allow  the  respondent  to  voice  any  usability  concerns  that  were  not  covered 
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in  the  Likert  style  questions,  and  second,  highlight  any  issues  the  study  may  have  omitted. 
The  responses  to  this  question  are  summarized  in  Table  2. 


“To  improve  the  usability  of  this  Web  site,  I  would” 

Type  of  Response 

Number  of  Responses 

Percent  of  Total 

No  Response 

16 

35% 

Written  Response 

30 

65% 

Table  2.  Comments  on  Usability  Improvements 


Of  the  30  responses,  most  commented  that  the  Web  site  was  user  friendly  and  did 
not  offer  suggestions  for  improvement  to  the  prototype.  The  responses  that  suggested 
changes  to  the  prototype  were  typified  by  the  comments  below. 


“Provide  more  links  on  the  first  page  to  make  it  more  efficient  and  easier 
to  the  final  page  I  am  after.  Index  or  reference  at  the  start  or  bottom  of  the 
page  would  work.” 

“Perhaps  on  the  opening  slide  there  could  be  something  like  a  table  of 
contents.  It  would  be  useful  to  know  everything  that  is  contained  on  the 
Web  site.” 

“...  Also,  home  page  is  confusing,  I  would  title  the  button  START  HERE 
instead  of  ENTER.” 

“Add  a  link  to  the  NASA  safety  site.” 

“Use  Frames.” 

A  common  theme  of  these  answers  was  one  of  mild  navigational  preferences,  some 
relating  to  verbage  and  others  related  to  the  physical  layout  of  the  Web  site. 


3.  User  Satisfaction  With  the  Online  MCAS 

Part  HI  of  the  Exit  Survey  contained  questions  pertaining  to  the  online  version  of 
the  MCAS.  To  effectively  use  the  online  MCAS,  the  user  needed  to  be  capable  of  finding 
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the  MCAS  in  the  Web  site,  and  he/she  also  needed  to  comprehend  the  method  of 
completing  the  questionnaire  and  submitting  it.  Questions  19  through  21  deal  with  the 
feelings  about  the  clarity  of  instructions  and  ease  of  completing  the  online  MCAS. 

Question  1 9  asks,  “Finding  the  online  MCAS  in  the  Aviation  Maintenance  Safety 
Web  site  was  straight  forward”  and  the  mean  was  4.44  (SD  =  .655).  The  range  for 
question  1 9  was  4,  and  the  histogram  of  the  frequency  distribution  is  presented  in 
Figure  19.  The  vast  majority  of  users  (96%)  either  strongly  agreed  or  agreed  with  this 
statement,  and  two  were  neutral  or  disagreed. 
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Strongly  Agree  Neutral  Disagree  Strongly 
Agree  Disagree 


Figure  19.  “Finding  the  MCAS” 

Question  20  asked,  “The  instructions  for  completing  the  MCAS  are  easily 
understood”  and  the  mean  was  4.64  (SD  -  .488).  The  range  for  question  20  was  2,  and 

i 

the  histogram  of  the  frequency  distribution  is  presented  in  Figure  20.  All  respondents 
(100%)  either  strongly  agreed  or  agreed  to  this  question. 
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Figure  20  “Instructions  for  MCAS  Understood” 


Question  21  asked,  “The  task  of  putting  answers  in  the  MCAS  was  easily 
completed”  and  the  mean  was  4.55  (SD  =  .585).  The  range  for  question  21  was  3,  and  the 
histogram  of  the  frequency  distribution  is  presented  in  Figure  21.  Mainly  users  (96%) 
strongly  agreed  or  agreed  with  this  question,  with  the  exception  of  two  persons  who  were 
neutral. 


e 

o 

G, 

03 

<D 


<D 

X 


3 


Agree  Disagree 


Figure  21.  “Submitting  Answers  to  the  MCAS  was  Easy” 
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The  final  question  in  Part  III  of  the  Exit  Survey  was  the  open  ended  question  “I 
would  make  these  changes  (if  any)  to  the  administration  of  the  online  MCAS.”  The 
response  rate  to  this  question  is  summarized  in  Table  3  below. 


“I  would  make  these  changes  (if  any)  to  the  administration  of  the  online 

MCAS.” 

Type  of  Response 

Number  of  Responses 

Percent  of  Total 

No  Response 

12 

26% 

Written  Response 

34 

74% 

Table  3.  Comments  on  MCAS 


Content  analysis  of  the  open-ended  responses  yielded  a  grouping  of  similar 
themes.  Users  commented  on  the  content  of  the  survey  as  well  as  the  administration, 
below  are  typical  responses  to  the  online  MCAS. 

“The  survey  is  too  long... Your  average  trooper  will  not  willingly  (hence 
honestly)  fill  out  such  an  in-depth  survey.” 

“I  am  a  Marine  53-E  Pilot  and  many  of  the  questions  did  not  pertain  to  me. 

Some  of  those  did  not  have  N/A  as  a  selection.” 

“E-6  Community-  Where  would  it  fall?” 

Administrative  Comments: 

“Have  an  option  of  no  graphics/cookies.  Units  at  sea  have  minimal 
bandwidth  and  download  time  is  terrible.” 

“I  tried  to  tab  thru  [sic]  the  questions  (instead  of  mouse).  I  thought  hitting 
ENTER  would  record  an  answer,  but  instead...  it  submitted  the  survey 
before  I  was  complete.” 
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having  to  manually  scroll  down  continually  to  answer  more  questions 
was  annoying.” 

“Have  more  questions  on  one  page” 

Comments  in  this  area  ranged  from  a  concern  that  the  length  of  the  survey  might 
inhibit  the  validity  of  some  respondent’s  answers  (attention  span)  to  physical 
limitations  of  the  movement  through  the  Web  site  (utilizing  back  button  of 
browser,  etc.). 

The  final  area  of  the  Exit  Survey  was  labeled  “Comments”  and  posed  two  the 
open  ended  questions  “The  most  negative  aspects  of  this  Web  site  were”  and  “The  most 
positive  aspects  of  this  Web  site  were”.  The  response  rate  to  each  of  these  questions  is 
summarized  in  Table  4  and  Table  5. 


“The  most  negative  aspects  of  this  Web  site  were.” 

Type  of  Response 

Number  of  Responses 

Percent  of  Total 

No  Response 

5 

11% 

Written  Response 

41 

89% 

Table  4.  Comments  on  Negative  and  Positive  Aspects  of  online  prototype 


“The  most  positive  aspects  of  this  Web  site  were.” 

Type  of  Response 

Number  of  Responses 

Percent  of  Total 

No  Response 

3 

7% 

Written  Response 

43 

93% 

Table  5.  Comments  on  Negative  and  Positive  Aspects  of  online  prototype 
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The  Comments  on  both  negative  and  positive  aspects  of  the  prototype  Web  site 
were  analyzed  and  the  most  representative  comments  are  listed  below. 

Negative  Aspects: 

“Having  to  back  out  after  completing  the  survey” 

“Marine  billets  not  included” 

“Lack  of  visual  stimulation  for  the  Gen  X/  Mountain  Dew  crowd.” 

“Too  much  wasted  space  on  the  main  pages” 

“Survey  might  be  too  long,  you’ll  surpass  the  attention  span  of  your  target 
audience.” 

“Survey  on  the  long  side” 

Positive  Aspects: 

“Ease  of  use,  simplicity” 

“Power  point  Lectures” 

“A  great  information  hwy  for  maintenance.  Nothing  but  positive  results 
can  come  from  this  site.” 

“Generally  easy  to  use” 

“Easy  access,  easy  to  navigate,  accessible  to  everybody” 
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“Should  provide  good  background  data” 

The  negative  comments  centered  on  a  concern  that  young  sailors  would  not  take 
the  time  to  complete  the  survey  because  of  it’s  length.  The  positive  comments 
characterized  a  Web  site  that  was  easy  to  use  and  contained  useful  information  for 
safety  programs. 

B.  ANOVA  TESTING 

1.  One  Way  ANOVA 

SPSS  software  provides  a  metric  called  the  significance  level,  also  known  as  p 
value,  which  allows  the  researcher  to  interpret  a  calculated  statistic  in  the  generated 
output  without  using  a  critical  value  table.  The  symbol  p  equates  to  the  probability  of  a 
Type  I  error.  A  Type  I  error  is  the  likelihood  of  rejecting  the  null  hypothesis  when  it  is 
actually  true.  If  the  reported  p  is  less  than  .05,  the  finding  is  statistically  significant  and 
the  null  Hypothesis  (no  difference)  is  rejected  (Morgan  &  Griego,  1998). 

A  one-way  ANOVA  was  used  to  compare  the  means  of  different  respondent 
experience  levels  to  test  for  significant  differences  in  levels  of  usability.  ANOVA 
assumes  the  Dependant  Variable  (DV)  is  interval  scale,  normally  distributed,  and  the 
variance  of  the  groups  are  equal.  If  the  assumptions  of  ANOVA  are  violated,  an 
alternative  nonparametric  test  called  the  Kruskal- Wallis  (K-W)  test  should  be  used  to 
compare  the  differences.  It  was  not  necessary  to  use  the  Kruskal-Wallis  test  with  the  data 
from  this  study,  since  it  met  all  the  parameters  necessary  to  for  ANOVA  analysis.  The 
Independent  Variable  (IV)  in  this  study  was  represented  as  ordinal  data  while  the  DVs 
were  measured  on  an  interval  scale.  The  distributions  of  each  of  the  DVs  were  compared 


50 


to  a  normal  distribution  curve  to  ensure  they  approximated  the  normal  distribution. 

The  equality  of  variances  was  tested  using  the  Levene  Test  (Kotz,  Johnson,  & 
Read,  1988).  Since  the  Levene  statistic  was  not  significant  (p  >.05),  the  assumption  of 
equal  variances  was  not  violated  and  the  K-W  test  was  not  needed.  (Morgan  &  Griego, 
1998).  The  ANOVA  statistic  compares  means  between  groups  and  is  used  to  identify  a 
statistically  significant  difference  between  means.  If  the  p  value  calculated  by  an 
ANOVA  test  is  significant  (p  <  .05)  then  there  is  a  significant  difference  between  groups 
with  respect  to  the  DV.  In  order  to  answer  the  difference  between  the  IV  and  the 
individual  DVs,  hypotheses  were  developed  and  tested. 

2.  Formulation  of  The  Null  Hypothesis 

The  null  hypothesis  is  used  for  testing;  it  is  a  statement  that  no  difference  exists 
between  the  parameter  and  the  statistic  being  compared  to  it  (Cooper  &  Emory,  1995). 
The  Alternative  hypothesis  is  the  logical  opposite  of  the  null  hypothesis.  If  statistically 
significant,  the  null  hypothesis  can  be  rejected,  and  the  alternative  hypothesis  is  accepted. 
The  following  hypotheses  formed  and  tested  to  compare  how  the  IV  (User  experience 
level)  impacted  the  DV  (User  impressions). 

1 )  IV:  How  long  have  you  been  using  an  Internet  Browser? 

DV:  I  found  the  Web  site  easy  to  Navigate. 

NULL  HYPOTHESIS:  there  is  no  difference  in  how  experience  level  impacted 
the  ease  of  navigation  of  the  Web  site. 

The  F  statistic  with  3,  and  42  degrees  of  freedom  was  1 .494.  This  resulted  in  a  p  value  of 
.234,  which  did  not  allow  rejection  of  the  null  hypothesis. 
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2)  IV:  How  long  have  you  been  using  an  Internet  Browser? 

DV:  I  feel  the  information  on  the  Web  site  was  in  a  logical  form. 

NULL  HYPOTHESIS:  There  is  no  difference  in  how  experience  level  impacted 
perception  of  whether  the  information  was  presented  in  a  logical  form. 

The  F  statistic  with  3,  and  42  degrees  of  freedom  was  .799.  This  resulted  in  a  p  value  of 
.503,  which  did  not  allow  rejection  of  the  null  hypothesis. 

3)  IV :  How  long  have  you  been  using  an  Internet  Browser? 

DV:  I  found  that  adequate  instructions  were  provided  to  navigate  the  Web  site. 
NULL  HYPOTHESIS:  There  was  no  difference  in  how  experience  level  impacted 
whether  respondent  felt  enough  instructions  were  provided. 

The  F  statistic  with  3,  and  42  degrees  of  freedom  was  1.494.  This  resulted  in  a  p  value  of 
.234,  which  did  not  allow  rejection  of  the  null  hypothesis. 

4)  IV:  How  long  have  you  been  using  an  Internet  Browser? 

DV:  It  was  easy  to  learn  to  use  this  Web  site. 

NULL  HYPOTHESIS:  There  is  no  difference  in  how  experience  level  impacted 
the  ease  of  learning  to  use  the  Web  site. 

The  F  statistic  with  3,  and  42  degrees  of  freedom  was  2.287.  This  resulted  in  a  p  value  of 
.097,  which  did  not  allow  rejection  of  the  null  hypothesis. 

With  each  case,  the  null  hypothesis  could  not  be  rejected.  The  comparison  of  the 
subject’s  experience  level  with  Internet  browsers  to  usability  concerns,  does  not  impact 
the  impression  the  subject  had  on  the  Web  site’s  usability. 
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V.  CONCLUSION 


A.  SUMMARY 

Naval  Aviation  is  posed  with  the  challenge  of  reducing  its  mishap  rate,  making  it 
essential  to  pursue  programs  and  processes  to  achieve  this.  Organizational  climate  has 
been  linked  to  risk  management  and,  consequently,  experienced  mishaps.  Understanding 
the  climate  that  pervades  an  organization  is  essential  if  changes  are  to  be  made  to 
ultimately  effect  the  mishap  rate. 

Many  theoretical  frameworks  have  been  developed  to  understand  organizational 
climate;  one  that  fits  the  characteristics  of  Naval  Aviation  is  the  High  Reliability 
Organization  model  (HRO).  Working  within  the  HRO  framework,  the  Command  Safety 
Assessment  (CSA)  survey  evaluated  the  safety  climate  within  Naval  Aviation  Squadrons 
from  aircrew  perspective.  The  Maintenance  Climate  Assessment  Survey  (MCAS)  was 
designed  to  measure  HRO  characteristics,  but  from  a  perspective  of  the  aviation 
maintainer. 

In  order  to  provide  for  an  effective  vehicle  to  distribute  the  MCAS,  the  World 
Wide  Web  (WWW)  was  explored.  The  WWW  offers  an  opportunity  to  obtain  a  wide 
distribution  of  material  at  a  very  small,  fixed  cost.  Shrinking  budgets  within  the  military 
necessitates  finding  new  methods  of  mission  accomplishment,  therefore  making  the 
efficiency  of  the  WWW  a  method  of  allowing  process  improvement  within  the  confines 
of  shrinking  budgets. 
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B.  CONCLUSIONS 


This  research,  based  on  the  initial  research  questions,  has  given  the  following: 

1)  Is  there  a  need  to  disseminate  and  administer  Web  based  materials  to  aviation 
units?  Naval  Aviation  is  challenged  to  meet  aggressive  reductions  in  its  mishap  rate  by 
the  year  2000.  Even  with  the  record  low  mishap  rates  that  have  been  achieved  in  recent 
years,  continued  improvement  is  necessary.  The  HFQMB  is  seeking  the  “invention”  of 
new  processes  and  procedures  to  help  meet  the  goals  set  forth  in  mishap  reduction.  A 
Web  based  MCAS  will  further  facilitate  climate  assessment  and  evaluation  throughout 
the  Naval  Aviation  Fleet,  improved  methods  of  assessment  will  lead  to  process 
improvements  for  Naval  Aviation. 

2)  Do  the  U.S.  Navy  and  Marine  Corps  squadrons  have  the  necessary 
software/hardware  capabilities  to  utilize  the  WWW?  As  technology  permeates  the  fleet, 
it  is  clear  that  the  ability  to  utilize  the  WWW  at  the  squadron  level  exists.  A  total  of  97 
percent  of  the  subjects  in  this  study  had  access  to  the  WWW  (n  =  46).  The  remaining  3 
percent  involved  in  this  study  were  not  familiar  with  the  computing  capability  at  their 
future  squadron  assignment. 

3)  Can  the  MCAS  be  effectively  distributed  and  administered  using  the  WWW? 
Development  software  available  enables  the  design  and  implementation  of  surveys  for 
applications  on  the  WWW.  The  HTML  editors  that  are  commonly  available  facilitate  the 
design  of  a  Web  site  for  distribution  of  the  MCAS.  The  current  level  of  technology  can 
easily  support  the  dissemination  and  collection  of  data  from  the  MCAS. 
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4)  Once  distributed,  do  personnel  have  the  necessary  ability  to  utilize  the 
electronic  version  of  the  MCAS?  The  subjects  in  this  study  were  representative  of  ASOs 
throughout  the  fleet,  82  percent  (n  =  46)  had  over  one  year  experience  using  the  Internet. 
Successfully  navigating  the  prototype  of  the  Naval  Aviation  Maintenance  Safety  Web 
Site,  subjects  not  only  demonstrated  the  requisite  knowledge  for  using  the  online  tool, 
they  demonstrated  an  acceptance  and  enthusiasm  for  the  project. 

Adequate  knowledge  exists  within  the  ASO  community  to  make  effective  use  of 
the  WWW  as  a  distribution  method  for  the  MCAS.  Using  the  efficiency  obtained 
through  the  WWW  allows  increased  distribution  of  the  MCAS  with  minimal  cost. 
Taking  advantage  of  the  software  and  hardware  that  is  currently  in  place  within  aviation 
squadrons  is  an  additional  step  to  achieving  the  improvements  that  Naval  Aviation  must 
accomplish  to  meet  its  goals  of  mishap  reduction. 

C.  RECOMMENDATIONS 

Many  areas  remain  open  for  further  development  and  testing  of  the  Naval 
Aviation  Maintenance  Safety  Web  Site.  It  is  suggested  that  further  research  efforts 
include  the  following  areas:  1)  Inclusion  of  survey  feedback  into  the  layout  and  design  of 
the  Web  site,  2)  Populating  the  pages  that  were  data  space  holders,  (e.g.,  summaries  of 
mishaps,  etc.),  and  3)  Further  development  of  data  capture  methods. 
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APPENDIX  A.  AVIATION  MAINTENANCE  SAFETY  WEB  SITE 


This  appendix  consists  of  three  parts:  (1)  A  verbal  description  of  the 
pages  that  make  up  the  Aviation  Maintenance  Safety  Web  site,  (2)  A  symbolic 
depiction  of  the  logical  layout  of  the  Web  site,  (3)  Screen  captures  of  the  pages 
that  make  up  the  complete  prototype  Aviation  Maintenance  Web  site. 

Part  1.  Verbal  Description  of  Web  Pages 

Page  1 :  Title  page  for  the  Web  site. 

Page  2:  Welcome/Purpose  page.  Branches  to  page:  2.1 

Page  2.1:  Safety  Survey  Page.  Branches  to  pages:  2.1.1,  2.1.2,  and  2.1.3 

Page  2.1.1:  MCAS 

Page  2.1.2:  Fleet  Statistics  on  Excel 

Page  2.1.3:  Interpretation  Materials 

Page  2.2:  Related  Web  sites. 

Page  3:  Aviation  Maintenance  Training  Page.  Branches  to  pages:  3.1,  3.2 

Page  3.1 :  Groundcrew  Coordination  Training.  Branches  to  pages:  3.1.1,  3.1.2,  and  3.1.3 

Page  3.1.1:  Workshop  Materials. 

Page  3.1.2:  Student  Guide. 

Page  3.1.3:  Case  Studies. 

Page  3.2:  FAA  Document. 

Page  4:  Community  Maintenance  Human  Factors  Briefs. 

Page  4.1 :  Community  Human  Factors  Briefs  for  Jets. 
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Page  4.2:  Community  Human  Factors  Briefs  for  Props. 
Page  4.3:  Community  Human  Factors  Briefs  for  Helos. 
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Part  2.  Symbolic  Depiction  of  the  Logical  Layout  of  the  Web  Site 
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Part  3.  Screen  Captures  of  the  Aviation  Maintenance  Safety  Web  Site 


Page  1 

Title  page  for  the  Web  site 
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Purpose:  The  purpose  of  this  survey  is  to  try'  and  gain  valuable  insight  into  the  maintenance  community's  perception  concerning 
aviation  mishaps  within  the  Navy  and  Marine  Corps.  Your  participation  and  answers  will  be  used  as  a  guide  in  the  Navy's  on-going 
effort  to  lower  the  aviation  mishap  rate. 

The  first  fifteen  questions,  part  I,  regard  biographical  data;  information  particular  to  yourself.  This  information  will  aid  in  the  analysis 
of  your  responses.  NO  attempts  will  be  made  to  identify  individual  respondents  or  their  organizations. 

Part  II  has  35  questions  pertaining  to  the  maintenance  community'.  Piease  respond  to  the  questions  with  the  answer  that  most 
correctly  reflects  your  honest  opinion. 

Thank  you  in  advance  forvour  participation! 

A.  Part  I 

1.  Your  rank? 
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APPENDIX  B.  INSTRUCTION  SHEET  FOR  SUBJECTS 


Instructions  for  viewing  the 

Aviation  Maintenance  Safety  Web  site 

These  instructions  will  assist  in  viewing  the  Aviation  Maintenance  Safety  Web  site. 


Computers  to  use: 

In  the  ASO  Ready  Room  3rd  Deck 
Computer  names: 

In  the  Computer  Lab  3rd  Deck 

Computer  names: 


Prowler 

Harrier 

Skywarrior 

Texan2 

Viking 

Cobra 


Accessing  the  Web  site: 

1 .  Log  on  the  computer  using  your  group  ASO  account 

2.  On  the  desktop,  double  click  on  the  icon  named  Aviation  Maintenance  Safety  Web 
site 


r^ 

G^ 

Aviation'"* 
Maintenance 
Safety  Web 
_ _ _crte _ “ 


3.  When  the  initial  page  of  the  Web  site  opens,  utilize  the  function  buttons  within 
Microsoft  Internet  Explorer  (Back,  Forward,  Stop,  etc.)  to  “surf”  through  the  Web  site. 


m 


Microsoft  Internet  Explorer  -  Microsoft  Internet  Explorer 
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APPENDIX  C.  TASK  LIST 


Aviation  Maintenance  Safety  Web  Site 
And 

Online  Version  of  the  Maintenance  Climate  Assessment  Survey  (MCAS) 


Purpose:  The  following  tasks  are  designed  to  introduce  a  developmental  version  of  the 
Aviation  Maintenance  Safety  Web  Site.  This  Web  site  is  being  created  to  assist  aviation 
squadron  safety  personnel  in  assessing  the  climate  of  safety  within  their  organization,  and 
to  provide  useful  safety  reference  material  for  squadron  training. 

Your  input  will  be  used  to  improve  this  Web  site  and  maximize  it's  utility  to  the  squadron 
safety  officer. 

Directions:  The  following  list  of  tasks  will  guide  your  "surf'  of  the  Aviation 
Maintenance  Safety  Web  Site.  Upon  completion  of  the  tasks,  a  short  questionnaire  will 
query  your  experiences  with  the  site.  There  is  room  for  comments  on  the  survey,  feel  free 
to  make  suggestions. 

Your  Assistance  in  making  this  future  Web  site  productive  is  greatly  appreciated! 


TASK  LIST 

As  you  complete  the  following  tasks,  make  a  mental  note 
of  any  difficulties  you  may  experience. 

1 .  During  your  tour  of  the  site,  find  the  online  version  of  the  Maintenance  Climate 
Assessment  Survey  (MCAS),  complete  the  survey  and  submit  your  results  using  the 
submit  button  at  the  bottom  of  the  form 


2.  Find  the  Naval  Safety  Center  homepage  through  the  Links  provided  in  the  Web  site 

3.  Search  the  Web  site  as  if  you  were  planning  squadron  training;  find  the  portion  of  the 
Web  Site  that  has  Microsoft  PowerPoint  presentations  to  download. 

4.  Find  the  area  of  the  Web  site  where  mishaps  by  aircraft  community  are  located. 
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APPENDIX  D.  EXIT  SURVEY 


User’s  impression  of  the 
Maintenance  Safety  Web  site  and 

Maintenance  Climate  Assessment  Survey  (MCAS) 


Purpose:  This  survey  evaluates  a  user’s  overall  satisfaction  of  the  Aviation  Maintenance  Safety  Web  site,  it  consists  of 
three  parts. 

Part  I:  Demographic  Information.  This  information  is  used  to  establish  the  user’s  computer  experience  and  determine 
the  software/hardware  being  used  in  the  fleet. 

Part  II:  User  satisfaction  with  the  Aviation  Maintenance  Safety  Web  site.  This  Section  deals  directly  with  participant 
experience  as  they  browsed  the  prototype  Maintenance  Safety  Web  site. 

Part  III:  User  Satisfaction  with  the  online  version  of  the  Maintenance  Climate  Assessment  Survey  (MCAS).  This 
section  allows  participants  to  comment  on  the  online  MCAS. 


Part  I.  Demographic  Information 

1 .  I  am  attached  to  a  command  that  performs  maintenance  at  the: 
(Choose  one  from  the  list) 

□  Squadron  Level 

□  Intermediate  Level 

□  Depot  Level 

□  None  of  the  Above 


2.  How  long  have  you  been  using  an  Internet  Browser? 
.(Choose  one  from  the  list) 


D  Less  than  one  month 

□  1  month  to  less  than  1  year 

□  1  year  to  less  than  2  years 

□  2  years  or  more 

3.  What  Internet  Browser  do  you  normally  use? 

(Select  all  that  apply  from  both  columns) 


At  Work  I  use:  At  Home  I  use: 

□  Netscape  Navigator  H 

□  Microsoft  Internet  Explorer 

□  AOL’s  version  of  Internet  Explorer 

□  Other  Browser 

□  None 


Netscape  Navigator 
I  Microsoft  Internet  Explorer 

[I  AOL’s  version  of  Internet  Explorer 

Z  Other  Browser 

□  None 


4.  What  categories  of  software  have  you  used  and  are  familiar  with? 
(Select  all  that  apply) 

□  Word  Processor  (MS  Word,  WordPerfect,  etc.) 

D  Spreadsheet  (MS  Excel,  Lotus  123,  etc.) 
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lJ  Graphic  Software  (Corel  Draw,  Adobe  PhotoShop,  etc.) 

□  Electronic  Mail  (AOL,  Eudora,  etc.) 

5.  What  computer  systems  do  you  use? 

(Select  all  that  apply) 

□  Windows/Windows  NT 

□  Macintosh 

□  UNIX 

□  Other 


PART  II 


USER  IMPRESSIONS 

(Fill  in  the  choice  that  matches  your  impression) 


Strongly 

Agree 

Agree 

Neutral  Disagree 

Strongly 

Disagree 

My  tour  of  the  Web  site 
Was  very  interesting 

□ 

□ 

C 

I  found  the  Web  site 
easy  to  Navigate 

□ 

□ 

—  p 

G 

I  feel  the  information  on 
the  Web  site  was  in  a 
logical  form 

r-; 

□ 

□  G 

□ 

The  information  presented 
in  the  Web  site  is  relevant 
to  maintenance  safety 

G 

□ 

G  □ 

G 

My  overall  impression  of  the  Web  site  is: 


During  my  tour  of  the  Web  site,  I  found: 


Strongly 

Agree 

Agree 

Neutral 

Disagree 

Strongly 

Disagree 

The  use  of  terms  consistent  □ 

□ 

□ 

□ 

□ 
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The  position  of  screen 


instructions  were  consistent  D 

□ 

u 

The  results  obtained  from 
the  Web  site  were  predictable  Z 

□ 

□ 

r-j 

□ 

That  adequate  instructions 
were  provided  to  navigate 
the  Web  site  G 

r 

G 

□ 

G 

I  feel  that: 

The  information  on  the 

Web  site  is  easy  to 

Understand  Z 

n 

□ 

i- 1 

D 

It  was  easy  to  learn  to 

use  this  Web  site  Z 

□ 

- 

□ 

□ 

The  organization  of  material 
on  this  Web  site  is  clear  □ 

r-j 

G 

□ 

□ 

I  could  effectively  use  the 
information  on  this  Web  site  Z 

G 

□ 

When  fully  implemented,  this 

Web  site  will  have  all  the 
functions  and  capabilities  I 
would  expect  it  to  have  G 

LJ 

r 

□ 

i-J 

To  improve  the  usability  of  this  Web  site,  I  would: 
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Part  III 


Online  MCAS 


Strongly  Agree  Neutral  Disagree  Strongly 

Agree  Disagree 


Finding  the  online  MCAS 
in  the  Aviation  Maintenance 
Safety  Web  site  was 
straight  forward 


The  instructions  for 
completing  the  MCAS 

are  easily  understood  □  □ 


The  task  of  putting  answers 
in  the  MCAS  form  was 

easily  completed  DC  □ 


I  would  make  these  changes  (if  any)  to  the  administration  of  the  online  MCAS 


Comments 

The  most  negative  aspects  of  this  Web  site  were: 


The  most  positive  aspects  of  this  Web  site  were: 


Your  participation  is  appreciated 
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